Keyboard gesture instruction generating method and computer program product and non-transitory computer readable storage medium thereof

ABSTRACT

A keyboard gesture instruction generating method includes: loading an instruction condition definition table, where the instruction condition definition table includes a plurality of gesture instructions, and each gesture instruction includes a plurality of instruction conditions, and each instruction condition has a corresponding defined value; reading an operating position image obtained by a sensor; converting the operating position image into an operating position data, where the operating position data includes relative position data of keys which are referenced; recognizing a gesture event according to multiple sets of the successive operating position data; obtaining a plurality of detected values of the gesture event and comparing the detected values with these defined values; if these detected values match with the defined values of all the instruction conditions of one of the gesture instructions, stopping tracking of the gesture event; and executing the matching gesture instruction.

CROSS-REFERENCE TO RELATED APPLICATION

This non-provisional application claims priority under 35 U.S.C. §119(a)to Patent Application No. 105113622 filed in Taiwan, R.O.C. on Apr. 29,2016, the entire contents of which are hereby incorporated by reference.

BACKGROUND Technical Field

The present invention relates to a keyboard operating method about acomputer, a mobile device, or a head-up display/head-mounteddisplay/virtual reality display device, and particularly to a method forgenerating a keyboard gesture operation instruction by using a physicalkeyboard and a sensor and a computer program product thereof.

Related Art

U.S. Pat. No. 8,928,590 B1 (inventor Tarek), of “GESTURE KEYBOARD METHODAND APPARATUS” discloses a keyboard which can capture an operationgesture. The keyboard captures an operation gesture of a user within agesture operation range (i.e., a visual range of a camera lens) by usinga pair of cameras, and may interact with a system after a gesture isrecognized.

The gesture operation range in the prior art is at a position about fourinches above the keyboard, and the operation gesture thereof does nothave any corresponding relationships to a key on a physical keyboard.Because an image captured by a camera does not use a key position as areferential basis, a problem that a gesture (i.e., an instruction) isdetermined erroneously may easily occur when performing imagerecognition. Therefore, it is necessary to improve the prior art toprovide a method for converting a gesture operation into a correspondinggesture instruction by using a key position on the physical keyboard asthe referential basis, so as to operate an application on a computingdevice, thereby avoiding a problem of erroneously determining a gesture.

SUMMARY

In view of the above, the present invention provides a keyboard gestureinstruction generating method and a computer program product and anon-transitory computer readable storage medium thereof, and anobjective thereof is generating a gesture instruction by using aphysical keyboard and a camera, so as to control an executed applicationon a computing device.

An embodiment of the present invention provides a keyboard gestureinstruction generating method which is applied to a computing deviceconnected to a physical keyboard, a screen and a sensor, the physicalkeyboard including a plurality of keys, and each of the keys respondingto a keystroke, so as to output a key scan code to the computing device,and the sensor being configured to detect and obtain an operatingposition image, where the keyboard gesture instruction generating methodincludes: loading an instruction condition definition table, where theinstruction condition definition table includes a plurality of gestureinstructions, and each gesture instruction includes a plurality ofinstruction conditions, and each instruction condition has acorresponding defined value; reading the operating position imageobtained by the sensor; converting the operating position image into anoperating position data, where the operating position data includesrelative position data of the keys which are referenced; recognizing agesture event according to multiple sets of the successive operatingposition data; obtaining a plurality of detected values of the gestureevent and comparing the detected values with these defined values; ifthese detected values match with the defined values of all theinstruction conditions of one of the gesture instructions, stoppingtracking of the gesture event; and executing the matching gestureinstructions.

The present invention again provides a computer program productcomprising a plurality of computer executable instructions stored in anon-transitory computer readable storage medium. The computer executableinstructions are loaded and executed by the computing device to causethe computing device to complete the above-described method forgenerating a keyboard gesture instruction.

The present invention further provides a non-transitory computerreadable storage medium storing a computer program, where the computerprogram includes a plurality of computer executable instructions forexecuting the foregoing method for generating a keyboard gestureinstruction.

The following implementation manners describe features and advantages ofthe present invention in detail. The content enables any person skilledin the art to know technical content of the present invention andimplement the technical content. A person skilled in the art can easilyunderstand the objectives and advantages of the present inventionaccording to the content, the application scope, and drawings disclosedin this specification.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram of a physical keyboard device according toa first embodiment of the present invention;

FIG. 2 is a schematic diagram of a physical keyboard device according toa second embodiment of the present invention;

FIG. 3 is a schematic diagram of a physical keyboard device according toa third embodiment of the present invention;

FIG. 4 is a schematic diagram of a physical keyboard device according toa fourth embodiment of the present invention;

FIG. 5 is a system block diagram of a physical keyboard device and acorresponding computing device according to a first embodiment of thepresent invention;

FIG. 6 is a system block diagram of a physical keyboard device and acorresponding computing device according to a fifth embodiment of thepresent invention;

FIG. 7 is a system block diagram of a physical keyboard device and acorresponding computing device according to a sixth embodiment of thepresent invention;

FIG. 8 is an operation flowchart of generating a gesture instructionaccording to an embodiment of the present invention;

FIG. 9 is a first schematic diagram of an on-screen keyboard accordingto a seventh embodiment of the present invention;

FIG. 10 is a second schematic diagram of an on-screen keyboard accordingto a seventh embodiment of the present invention;

FIG. 11 is a third schematic diagram of an on-screen keyboard accordingto a seventh embodiment of the present invention;

FIG. 12 is a fourth schematic diagram of an on-screen keyboard accordingto a seventh embodiment of the present invention;

FIG. 13 is a fifth schematic diagram of an on-screen keyboard accordingto a seventh embodiment of the present invention;

FIG. 14 is a sixth schematic diagram of an on-screen keyboard accordingto a seventh embodiment of the present invention;

FIG. 15 is a seventh schematic diagram of an on-screen keyboardaccording to a seventh embodiment of the present invention;

FIG. 16 is a first schematic diagram of an on-screen keyboard accordingto an eighth embodiment of the present invention;

FIG. 17 is a second schematic diagram of an on-screen keyboard accordingto an eighth embodiment of the present invention;

FIG. 18 is a third schematic diagram of an on-screen keyboard accordingto an eighth embodiment of the present invention;

FIG. 19 is a schematic diagram of an on-screen keyboard according to aninth embodiment of the present invention;

FIG. 20 is a schematic diagram of an on-screen keyboard according to atenth embodiment of the present invention;

FIG. 21 is an operation flowchart of an on-screen keyboard according toan embodiment of the present invention; and

FIG. 22 is a schematic diagram of an on-screen keyboard and a gestureoperation track according to an embodiment of the present invention.

DETAILED DESCRIPTION

The present invention is applied to but not limited to computing devicessuch as, a personal computer (PC/Mac), a notebook computer(Laptop/Portable Computer), a Smartphone, a tablet PC, a Smart TV, ahead-up display/head-mounted display/virtual reality display device, anda video game console or TV game console, etc. which are in communicationconnections to physical keyboards 10, 20, 30, 40, and 50 and screens 76a, 76 b, and 76 c. The communication connection manner may be a wirelessor wired communication manner. The physical keyboards 10, 20, 30, 40,and 50 and the screens 76 a, 76 b, and 76 c may be disposed outside thecomputing device and to be connected thereto, or may be disposed as apart of the computing device.

The accompanying drawings of the present description are used todescribe the functions and technical features of the present invention,and are not intended to limit the appearance of the present invention. Aplurality of embodiments of the physical keyboards 10, 20, 30, 40, and50 and a plurality of embodiments of on-screen keyboard 68 a, 68 b, 68c, and 68 d are respectively described below. The physical keyboards 10,20, 30, 40, and 50 of these different embodiments and the on-screenkeyboard 68 a, 68 b, 68 c, and 68 d of these different embodiments maycooperate with each other arbitrarily.

Referring to FIG. 1, FIG. 1 is a schematic diagram of a physicalkeyboard according to a first embodiment of the present invention. Aphysical keyboard 10 in this embodiment includes a base 11, a pluralityof keys 12, a light source module 13, a first sensor 14, a second sensor15, and a control module 16. The base 11 includes an accommodating space111 for accommodating the plurality of keys 12. The physical keyboard 10shown in FIG. 1 accommodates the keys 12 having different functions byusing the openings of multiple areas. This is an existing commontechnical means, and is not described in detail herein.

The light source module 13 is disposed on the base 11, and is configuredto provide a light source for the plurality of keys 12. As shown in FIG.1, a holder 112 for accommodating the light source module 13 is disposedon an outer side at a central position of an upper longer side of thebase 11. The light source module 13 is disposed facing the plurality ofkeys 12, to provide a light source required by the plurality of keys 12.In this embodiment, to provide the light source for all of the keys 12,a height at which the light source module 13 is disposed is higher thanthose of all of the keys, so that the light source emitted by the lightsource module 13 can be irradiated onto all keys 12, or a sufficientambient luminance is generated in the areas in which all of the keys 12are located. In addition, the light source module 13 may be an LEDelement, and can emit visible light. In another implementation aspect,the light source module may be an LED element capable of emittinginfrared light. No limitation is imposed herein in the presentinvention.

The first sensor 14 and the second sensor 15 are also disposed on thebase 11, and face the plurality of keys 12. In this embodiment, thefirst sensor 14 and the second sensor 15 are also disposed on the holder112, and face the plurality of keys 12. The first sensor 14 isconfigured to detect an ambient luminance of the plurality of keys 12.The second sensor 15 is configured to detect and obtain an “operatingposition image”. In this embodiment, the holder 112 is disposed on theouter side at the central position of the upper longer side of the base11, so that the light source module 13, the first sensor 14, and thesecond sensor 15 can provide the light source for the plurality of keys12 in a left-right symmetric manner or perform detection in a left-rightsymmetric manner. However, no limitation is imposed herein in thepresent invention. In another implementation aspect, the holder 112 maybe disposed at any position on any side of the base 11, as long as alight source can be provided for the plurality of keys 12, and theplurality of keys 12 can be detected or irradiated. In addition, in thisembodiment, the light source module 13, the first sensor 14, and thesecond sensor 15 are all disposed on the same holder 112. However, nolimitation is imposed herein in the present invention. The light sourcemodule 13, the first sensor 14, and the second sensor 15 may also berespectively disposed at any position of the base 11, or every two maybe disposed at any position of the base 11. Alternatively, the firstsensor 14 is disposed at any position of the base 11, and the lightsource module 13 and the second sensor 15 are disposed on the sameholder 112.

In this embodiment, the first sensor 14 is a luminance sensor, orreferred to as an ambient light sensor. The first sensor 14 can detect aluminance of an environment in which the plurality of keys 12 islocated. The first sensor 14 is configured to determine whether theluminance of the environment in which the plurality of keys 12 islocated is sufficient, to enable the second sensor 15 to detect andobtain required data.

In this embodiment, the second sensor 15 is a camera (or a cameramodule). The operating position image detected and obtained by thesecond sensor 15 is an image, which is a photo image of a finger(fingers) and keys 12, obtained by photographing. The image istransmitted to a computing device 64 a (as shown in FIG. 5) by thecontrol module 16. An image analyzing module 58 disposed on thecomputing device 64 a analyzes the image, to obtain the “operatingposition data”, where the operating position data includes a relativeposition data of a key 12 which is touched or is approached (these twoare referred to as “referenced”), that is, “the data of relativepositions between a finger (or fingers) of a user and the keys 12”.Because the second sensor 15 can correctly determine the data ofrelative positions between a finger (or fingers) of a user and the keys12 only if all of the keys 12 are photographed, a position at which thesecond sensor 15 is disposed needs to be higher than a surface of anykey 12.

The control module 16 is disposed in the accommodating space 111 of thebase 11, and electrically connected to the plurality of keys 12, thelight source module 13, the first sensor 14, and the second sensor 15.The control module 16 may be located at any position in theaccommodating space 111 of the base 11. No limitation is imposed hereinin the present invention. When receiving the ambient luminance detectedby the first sensor 14 and determining that the ambient luminancedetected by the first sensor 14 is equal to or greater than a presetambient luminance, the control module 16 turns on the second sensor 15and receives the operating position image of the second sensor 15. Whendetermining that the ambient luminance detected by the first sensor 14is less than the preset ambient luminance, the control module 16 turnson the light source module 13 to provide the light source for theplurality of keys 12.

The preset ambient luminance herein is an ambient luminance at which thesecond sensor 15 can clearly detect and obtain the operating positionimage. The preset ambient luminance may be specified in advance, andthen pre-stored in the control module 16 to facilitate subsequentcomparison and determining. The operating position image received by thecontrol module 16 needs to be analyzed by the computing device 64 a.

Before starting to obtain the operating position image, the controlmodule 16 first determines whether the ambient luminance detected by thefirst sensor 14 is equal to or greater than the preset ambientluminance. If the ambient luminance detected by the first sensor 14 isequal to or greater than the preset ambient luminance, it indicates thatthe second sensor 15 can clearly detect and obtain the operatingposition image. In this case, the second sensor 15 is turned on to startphotographing and detect and obtain the operating position image. Whenthe control module 16 determines that the ambient luminance detected bythe first sensor 14 is less than the preset ambient luminance, itindicates that the image photographed by the second sensor 15 may beexcessively dark from which the operating position image cannot bedetected and obtained, so that the control module 16 turns on the lightsource module 13 to provide the light source for the plurality of keys12, to improve the ambient luminance of the areas in which the pluralityof keys 12 is located.

In this way, the foregoing physical keyboard 10 can resolve the problemof the insufficient ambient light source by using an automaticsupplemental lighting technology.

When the foregoing physical keyboard 10 is applied to a head-up display,a head-mounted display, or a virtual reality display (VR) device, evenif in a dark environment in which no lamp is turned on, a user can stilltype data with a physical keyboard 10 by means of the automaticsupplemental lighting technology in the this embodiment.

In this embodiment, when the second sensor 15 is already turned on,keeps photographing, and detects and obtains the operating positionimage, the control module 16 can know whether the operating positionimage includes “the data of relative positions between a finger (orfingers) and the plurality of keys 12” (refer to description of FIG. 5below for details).

When the operating position image does not include the “the data ofrelative positions between a finger (or fingers) and the plurality ofkeys 12”, it indicates that the user does not continue operating thephysical keyboard 10. In this case, the control module 16 turns off thesecond sensor 15, and the second sensor 15 stops photographing ordetecting and obtaining the operating position image. In this way, thesecond sensor 15 may be turned off automatically when the user does notoperate the physical keyboard 10, or has not operated the physicalkeyboard 10 for an idle timeout, to save power.

In addition, when the light source module 13 is in a turn-on state, thefirst sensor 14 keeps detecting an ambient luminance. When the ambientluminance detected by the first sensor 14 is greater than the presetambient luminance, the control module 16 turns off the light sourcemodule 13. In this case, the ambient luminance has been improved, forexample, the user has turned on an indoor lamp, and the ambientluminance detected by the first sensor 14 is greater than the presetambient luminance, so that assistance from the light source of the lightsource module 13 is not needed. In this case, the control module 16automatically turns off the light source module 13, to save power.

Next, referring to FIG. 2, FIG. 2 is a schematic diagram of a physicalkeyboard 20 according to a second embodiment of the present invention.Elements and structures in this embodiment that are the same as those inthe first embodiment are represented by using the same element symbols,and are not described in detail herein again. This embodiment isdifferent from the first embodiment in that a physical keyboard 20 inthis embodiment includes a plurality of light source modules, that is,light source modules 23, and the plurality of light source modules 23 isdisposed in an accommodating space 111 of a base 11 and below theplurality of keys 12, and faces the plurality of keys 12, to provide alight source.

The plurality of light source modules 23 in this embodiment is aplurality of LED elements 231 disposed on a circuit board 232 and belowthe keys 12. In another implementation aspect, the plurality of lightsource modules may further be LED elements disposed beneath each keycap,or may be a plurality of LED elements disposed on a side of theaccommodating space of the base, and enables, in cooperation withelements such as a light guide plate and a reflective plate, a lightsource to be emitted from the accommodating space toward the keys 12.

In this embodiment, when the control module 16 determines that anambient luminance detected by the first sensor 14 is less than a presetambient luminance, the control module 16 successively turns on at leastone of the plurality of light source modules 23, until the controlmodule 16 determines that the ambient luminance detected by the firstsensor 14 is equal to or greater than the preset ambient luminance, orthe plurality of light source modules 23 is all turned on. That is, whendetermining that the current ambient luminance is insufficient, thecontrol module 16 may first control one or more light source modules 23to be turned on, to provide the light source. Next, the control module16 keeps receiving an ambient luminance detected by the first sensor 14and determining the ambient luminance detected by the first sensor 14,and if the ambient luminance is still insufficient, the control module16 further turns on one or more light source modules 23. The controlmodule 16 does not stop further enabling light source modules 23 untildetermining that the ambient luminance detected by the first sensor 14is equal to or greater than the preset ambient luminance. Alternatively,the control module 16 successively turns on light source modules 23until all light source modules 23 are turned on.

In this way, the control module 16 determines and turns on only lightsource modules 23 sufficient for providing a required ambient luminancewithout simultaneously turning on all light source modules 23 each timean ambient luminance is insufficient. Therefore, the physical keyboard20 in this embodiment only needs to turn on a sufficient number of lightsource modules 23, thereby achieving an effect of power saving.

Referring to FIG. 3, FIG. 3 is a schematic diagram of a physicalkeyboard 30 according to a third embodiment of the present invention. Aphysical keyboard 30 in this embodiment includes a first keyboard 31, asecond keyboard 32, a first light source module 33, a second lightsource module 34, a first sensor 35, a second sensor 36, a third sensor37, and a control module 38. The first keyboard 31 includes: a firstbase 311 that includes a first accommodating space 312, and a pluralityof first keys 313. The plurality of first keys 313 is disposed in thefirst accommodating space 312. The second keyboard 32 includes: a secondbase 321 that includes a second accommodating space 322, and a pluralityof second keys 323. The plurality of second keys 323 is disposed in thesecond accommodating space 322. In this embodiment, the first keyboard31 and the second keyboard 32 are respectively a left keyboard and aright keyboard that can be used in pair and that are formed by dividingkeys on a common keyboard into two groups. However, no limitation isimposed herein in the present invention. Alternatively, the firstkeyboard and the second keyboard may be a combination of a letter symbolkeyboard and a numeric keyboard.

The first light source module 33 is disposed on the first base 311, andis configured to provide a first light source for the plurality of firstkeys 313. As shown in FIG. 3, a first holder 314 for accommodating thefirst light source module 33 is disposed on an outer side at a centralposition of an upper longer side of the first base 311. The first lightsource module 33 is disposed facing the plurality of first keys 313, toprovide the light source required by the plurality of first keys 313. Inthis embodiment, to provide the light source for all of the first keys313, a height of a position of the first light source module 33 ishigher than those of all keys, so that the light source emitted by thefirst light source module 33 can be irradiated onto all of the firstkeys 313, or a sufficient ambient luminance can be generated in areas inwhich all of the first keys 313 are located.

Similarly, the second light source module 34 is disposed on the secondbase 321, and is configured to provide a second light source for theplurality of second keys 323. As shown in FIG. 3, a second holder 324for accommodating the second light source module 34 is disposed on anouter side at a central position of an upper longer side of the secondbase 321. The second light source module 34 is disposed facing theplurality of second keys 323, to provide the light source required bythe plurality of second keys 323. In this embodiment, to provide thelight source for all of the second keys 323, a height of a position ofthe second light source module 34 is higher than those of all of thekeys, so that the light source emitted by the second light source module34 can be irradiated onto all of the second keys 323, or a sufficientambient luminance can be generated in areas in which all of the secondkeys 323 are located.

The first sensor 35 is disposed on the first base 311, and is configuredto detect an ambient luminance of the plurality of first keys 313. Asshown in FIG. 3, the first sensor 35 is disposed on the first holder314, and faces the plurality of first keys 313. Because the first sensor35 does not need to face all of the keys, and only needs to detect asingle point, in other embodiments, the first sensor 35 may be disposedat another position of the first base 311 or the second base 321, forexample, in a plane space (less than a height of a key) near any key. Inaddition, the first keyboard 31 is usually not excessively distant fromthe second keyboard 32, and an ambient luminance of the first keyboard31 is usually not excessively different from the second keyboard 32.Therefore, in this embodiment, the first sensor 35 (detecting only theambient luminance of the first keyboard 31) enables the control module16 to simultaneously control the first light source module 33 and thesecond light source module 34.

In this embodiment, the first sensor 35 is a luminance sensor, orreferred to as an ambient light sensor. The first sensor 35 can detect aluminance of an environment in which the plurality of first keys 313 islocated. In other embodiments, the first sensor 35 is disposed on thesecond base 321 of the second keyboard 32, to detect a luminance of anenvironment in which the plurality of second keys 323 is located. Thefirst sensor 35 is configured to determine whether the luminance of theenvironment, in which the plurality of first keys 313 (when disposed onthe first keyboard 31) or the plurality of second keys 323 (whendisposed on the second keyboard 32) is located, is sufficient, to enablethe second sensor 36 and the third sensor 37 to detect and obtainrequired data.

The second sensor 36 is disposed on the first base 311, and faces theplurality of first keys 313. In this embodiment, both the second sensor36 and the first light source module 33 are disposed on the first holder314, and face the plurality of first keys 313. The second sensor 36 isconfigured to detect and obtain first operating position image. In thisembodiment, the first holder 314 is disposed on the outer side of thecentral position on an upper side of the first base 311, so as to enablethe first light source module 33 and the second sensor 36 to provide thelight source for the plurality of first keys 313 in a left-rightsymmetric manner or perform detection in a left-right symmetric manner.However, no limitation is imposed herein in the present invention. Inanother implementation aspect, the first holder 314 may be disposed atany position on any side of the first base 311, as long as the lightsource can be provided for the plurality of first keys 313, and theplurality of first keys 313 can be detected and irradiated. In addition,in this embodiment, both the first light source module 33 and the secondsensor 36 are disposed on the same first holder 314. However, nolimitation is imposed herein in the present invention. The first lightsource module 33 and the second sensor 36 may further be respectivelydisposed at any position of the first base 311.

The third sensor 37 is disposed on the second base 321, and faces theplurality of second keys 323. In this embodiment, both the third sensor37 and the second light source module 34 are disposed on the secondholder 324, and face the plurality of second keys 323. The third sensor37 is configured to detect and obtain second operating position image.In this embodiment, the second holder 324 is disposed on the outer sideof the central position on the upper side of the second base 321, sothat the second light source module 34 and the third sensor 37 canprovide the light source for the plurality of second keys 323 in aleft-right symmetric manner or perform detection in a left-rightsymmetric manner. However, no limitation is imposed herein in thepresent invention. In another implementation aspect, the second holder324 may be disposed at any position on any side of the second base 321,as long as it is capable of providing the light source for the pluralityof second keys 323, and the plurality of second keys 323 can be detectedand irradiated. In addition, both the second light source module 34 andthe third sensor 37 in this embodiment are disposed on the same secondholder 324. However, no limitation is imposed herein in the presentinvention. The second light source module 34 and the third sensor 37 mayfurther be respectively disposed at any position of the second base 321.

In this embodiment, the second sensor 36 and the third sensor 37 arecameras (or camera modules). The operating position image detected andobtained by the second sensor 36 and the operating position imagedetected and obtained by the third sensor 37 are respectively a firstimage and a second image that are obtained by photographing. The firstimage and the second image are transmitted to a computing device 64 a bythe control module 38. An image analyzing module 58 disposed on thecomputing device 64 a analyzes the images, to obtain the operatingposition data (that is, “data of relative positions between afinger/fingers of a user and the plurality of first keys 313”, or “dataof relative positions between a finger/fingers of a user and theplurality of second keys 323”). Because the second sensor 36 cancorrectly determine the “data of relative positions between afinger/fingers of a user and the plurality of first keys 313” only whenall first keys 313 are photographed, and the third sensor 37 cancorrectly determine the “data of relative positions between afinger/fingers of a user and the plurality of second keys 323” only whenall second keys 323 are photographed, a position of the second sensor 36needs to be higher than a surface of any one of the first keys 313, anda position of the third sensor 37 needs to be higher than a surface ofany one of the second keys 323.

The control module 38 is disposed in the second accommodating space 322,and is electrically connected to the plurality of first keys 313, theplurality of second keys 323, the first light source module 33, thesecond light source module 34, the first sensor 35, the second sensor36, and the third sensor 37. Although the control module 38 herein islocated in the second accommodating space 322, no limitation is imposedherein in the present invention.

However, in another implementation aspect, the control module mayfurther be disposed in the first accommodating space. In addition,although the control module 38 in this embodiment is disposed in thesecond accommodating space 322, the control module 38 can control, byusing a wireless module, each element in the first keyboard 31 to beturned on or off, and can receive data detected and obtained by eachelement in the first keyboard 31. In another implementation aspect, thefirst keyboard and the second keyboard may further be connected in awired manner. No limitation is imposed herein in the present invention.In addition, in some implementation aspects, a first control module maybe disposed in the first keyboard, and a second control module may bedisposed in the second keyboard, to respectively control operation ofeach element in the first keyboard and the second keyboard.

When determining that the ambient luminance detected by the first sensor35 is equal to or greater than a preset ambient luminance, the controlmodule 38 turns on the second sensor 36 and the third sensor 37, andreceives the first operating position image of the second sensor 36 andthe second operating position image of the third sensor 37. Whendetermining that the ambient luminance detected by the first sensor 35is less than the preset ambient luminance, the control module 38 turnson the first light source module 33 to provide the first light sourcefor the plurality of first keys 313, or turns on the second light sourcemodule 34 to provide the second light source for the plurality of secondkeys 323, or simultaneously turns on the first light source module 33and the second light source module 34.

The preset ambient luminance herein is an ambient luminance at which thesecond sensor 36 and the third sensor 37 can clearly detect and obtainthe operating position image. The preset ambient luminance may bespecified in advance, and pre-stored in the control module 38 tofacilitate subsequent comparison and determining. The operating positionimages received by the control module 38 need to be analyzed by thecomputing device 64 a.

Before starting to obtain the first operating position image and thesecond operating position image, the control module 38 needs to firstdetermine whether the ambient luminance detected by the first sensor 35is equal to or greater than the preset ambient luminance. If the ambientluminance detected by the first sensor 35 is equal to or greater thanthe preset ambient luminance, it indicates that the second sensor 36 andthe third sensor 37 can clearly detect and obtain the operating positionimages. In this case, the control module 38 turns on the second sensor36 and the third sensor 37 to start photographing and detect and obtainthe first operating position image and the second operating positionimage. When the control module 38 determines that the ambient luminancedetected by the first sensor 35 is less than the preset ambientluminance, it indicates that the image photographed by the second sensor36 or the third sensor 37 may be excessively dark from which the firstoperating position image or the second operating position image cannotbe detected and obtained. In this case, the control module 38 turns onthe first light source module 33 to provide a light source for theplurality of first keys 313, or turns on the second light source module34 to provide a light source for the plurality of second keys 323, orsimultaneously turns on the first light source module 33 and the secondlight source module 34, to improve the ambient luminance of areas inwhich the plurality of first keys 313 and the plurality of second keys323 are located.

In this way, the foregoing structure can resolve the problem of theinsufficient ambient light source by using an automatic supplementallighting technology on the left and right split keyboards. When theforegoing structure is applied to a head-up display, a head-mounteddisplay, or a virtual reality display (VR) device, even if in a darkenvironment in which no lamp is turned on, a user can still type datawith the physical keyboard 30 by means of the automatic supplementallighting technology in the this embodiment.

In this embodiment, after the second sensor 36 and the third sensor 37are already turned on, keep photographing, and detect and obtain thefirst operating position image and the second operating position image,the control module 38 can learn whether the first operating positionimage includes the “data of relative positions between a finger/fingersand the plurality of the first keys 313”, and whether the secondoperating position image includes the “data of relative positionsbetween a finger/fingers and the plurality of the second keys 323”. Thefirst and second operating position data may be analyzed by thecomputing device 64 a and then, transmitted back to the control module38. Reference may be made to the descriptions about FIG. 5 in thefollowing for details. Although the device in FIG. 5 is not the deviceof this embodiment, the principles are substantially the same. Whenneither the first operating position image nor the second operatingposition image includes the “data of relative positions between afinger/fingers and the plurality of the first keys 313” and the “data ofrelative positions between a finger/fingers and the plurality of thesecond keys 323”, it indicates that the user does not continue operatingthe physical keyboard 30. In this case, the control module 38 turns offthe second sensor 36 and the third sensor 37 and does not continuephotographing or detecting and obtaining first operating position imageand second operating position image. In this case, the second sensor 36and the third sensor 37 may be turned off automatically when the userdoes not operate the physical keyboard 30, or has not operated thephysical keyboard 30 for an idle timeout, to save power.

In addition, when the first light source module 33 or the second lightsource module 34 is in a turn-on state, and the ambient luminancedetected by the first sensor 35 is equal to or greater than the presetambient luminance, the control module 38 turns off the first lightsource module 33 or the second light source module 34. That is, when thefirst light source module 33 is in a turn-on state, or the second lightsource module 34 is in a turn-on state, or both the first light sourcemodule 33 and the second light source module 34 are in a turn-on state,the first sensor 35 keeps detecting an ambient luminance. When theambient luminance detected by the first sensor 35 is greater than thepreset ambient luminance, the control module 38 turns off the firstlight source module 33 and the second light source module 34. In thiscase, the ambient luminance has been improved, for example, the user hasturned on an indoor lamp, and the ambient luminance detected by thefirst sensor 35 is greater than the preset ambient luminance, so thatassistance from the light source of the first light source module 33 orthe second light source module 34 is not needed. In this case, thecontrol module 38 automatically turns off the first light source module33 and/or the second light source module 34, to save power.

Subsequently, referring to FIG. 4, FIG. 4 is a schematic diagram of aphysical keyboard 40 according to a fourth embodiment of the presentinvention. Elements and structures in this embodiment that are the sameas those in the third embodiment are represented by using the sameelement symbols, and are not described in detail herein again. Thisembodiment is different from the third embodiment in that a physicalkeyboard 40 in this embodiment includes a plurality of first lightsource modules, that is, first light source modules 43, and theplurality of first light source modules 43 is disposed in a firstaccommodating space 312 of a first base 311 and below the plurality offirst keys 313 and can provide a first light source toward the pluralityof first keys 313. In addition, there is a plurality of second lightsource modules, that is, second light source modules 44, and theplurality of second light source modules 44 is disposed in a secondaccommodating space 322 of the second base 321 and below the pluralityof second key 323, and can provide a second light source toward theplurality of second keys 323.

The plurality of first light source modules 43 in this embodiment is aplurality of LED elements 431 that is disposed on a circuit board 432and below the first keys 313. The plurality of second light sourcemodules 44 is a plurality of LED elements 441 that is disposed on acircuit board 442 and below the second keys 323. In anotherimplementation aspect, the plurality of light source modules may furtherbe LED elements disposed beneath each keycap, or may be a plurality ofLED elements disposed on a side of the accommodating space of the base,and enables, in cooperation with elements such as a light guide plateand a reflective plate, a light source to be emitted from theaccommodating space toward the keys.

In addition, this embodiment is different from the third embodiment inthat there are two first sensors 45. One of the first sensors 45 isdisposed on the first base 311, and is, together with the second sensor36, disposed on the first holder 314 in this embodiment. The other firstsensor 45 is disposed on the second base 321, and is, together with thethird sensor 37, disposed on the second holder 324 in this embodiment.

Further, first sensors 45 for detecting the plurality of first keys 313and the plurality of second keys 323 are respectively disposed on thefirst keyboard 31 and the second keyboard 32 to detect an ambientluminance of the first keyboard 31 and an ambient luminance of thesecond keyboard 32, to obtain an ambient luminance value of eachkeyboard. In this way, the first light source module 43 and the secondlight source module 44 may be respectively controlled to be turned on oroff, to obtain a required ambient luminance, so that the first lightsource module 43 and the second light source module 44 do not need to besimultaneously turned on, thereby achieving an effect of power saving.

In this embodiment, when determining that the ambient luminance detectedby the first sensor 45 located on the first base 311 is less than apreset ambient luminance, the control module 38 successively turns on atleast one of the plurality of first light source modules 43 until thecontrol module 38 determines the ambient luminance detected by the firstsensor 45 is equal to or greater than the preset ambient luminance, oruntil the plurality of first light source modules 43 is all turned on.That is, when determining that the current ambient luminance isinsufficient, the control module 38 may first control one first lightsource module 43 to be turned on (controlling an LED element 431 to beturned on herein), to provide a light source. Subsequently, the controlmodule 38 keeps receiving and determining the ambient luminance detectedby the first sensor 45, and if the ambient luminance is stillinsufficient, the control module 38 further turns on a first lightsource modules 43 (controlling an LED element 431 to be turned onherein). The control module 38 does not stop further turning on firstlight source module 43 until determining that the ambient luminancedetected by the first sensor 45 is equal to or greater than the presetambient luminance. Alternatively, the control module 38 successivelyturns on first light source modules 43 until all of the first lightsource modules 43 are turned on. Although an example is used, in whichone LED element is turned on at a time, for description herein, inanother implementation aspect, two or more LED elements may be groupedinto one group, so that two or more LED elements 431 can be turned on ata time.

Similarly, when determining that the ambient luminance detected by thefirst sensor 45 located on the second base 321 is less than the presetambient luminance, the control module 38 successively turns on at leastone of the plurality of second source modules 44, until the controlmodule 38 determines that the ambient luminance detected by the firstsensor 45 is equal to or greater than the preset ambient luminance, orthe plurality of second light source modules 44 is all turned on. Thatis, when determining that the current ambient luminance is insufficient,the control module 38 may first control a second light source modules 44to be turned on (controlling an LED element 441 to be turned on herein),to provide a light source.

Subsequently, the control module 38 keeps receiving and determining theambient luminance detected by the first sensor 45. If the ambientluminance is still insufficient, the control module 16 further turns ona second light source module 44 (controlling an LED element 441 to beturned on herein). The control module 38 does not stop further turningon a second light source module 44 until determining that the ambientluminance detected by the first sensor 45 is equal to or greater thanthe preset ambient luminance. Alternatively, the control module 38successively turns on second light source modules 44 until all of thesecond light source modules 44 are turned on. Although an example isused, in which one LED element 441 is turned on at a time, fordescription herein, in another implementation aspect, two or more LEDelements 441 may be grouped into a group, so that two or more LEDelements 441 can be turned on at a time.

In this way, the control module 38 performs determination to turn ononly first light source modules 43 or second light source modules 44sufficient for required ambient luminance, and does not need tosimultaneously turn on all first light source modules 43 or second lightsource modules 44 each time the ambient luminance is insufficient.Therefore, the physical keyboard 40 in this embodiment not only can turnon sufficient first light source modules 43 or second light sourcemodules 44 when necessary, but also can achieve an effect of powersaving.

In addition, because there are two first sensors 45 that arerespectively located on the first base 311 and the second base 321 inthis embodiment, the ambient luminance of a plurality of first keys 313and the ambient luminance of the second key 323 can be respectivelydetected.

Therefore, when the first light source module 43 is in a turn-on state,and the ambient luminance detected by the first sensor 45 located on thefirst base 311 is equal to or greater than the preset ambient luminance,the control module 38 turns off the first light source module 43.Alternatively, when the second light source module 44 is in a turn-onstate, and the ambient luminance detected by the first sensor 45 locatedon the second base 321 is equal to or greater than the preset ambientluminance, the control module 38 turns off the second light sourcemodule 44.

That is, when the first light source module 43 is in a turn-on state, orthe second light source module 44 is in a turn-on state, or both thefirst light source module 43 and the second light source module 44 arein a turn-on state, the two first sensors 45 keep detecting an ambientluminance of the first keyboard 31 and an ambient luminance of thesecond keyboard 32, and respectively turns off the first light sourcemodule 43 or the second light source module 44 according to the ambientluminance detected by the first sensors 45. In this case, the ambientluminance has been improved, for example, the user has turned on anindoor lamp, and the ambient luminance detected by the first sensor 45is greater than the preset ambient luminance, so that assistance fromthe light source of the first light source module 33 or the second lightsource module 34 is not needed. In this case, the control module 38automatically turns off the first light source module 43 or the secondlight source module 44, to save power.

Referring to FIG. 5, FIG. 5 is a system block diagram of the physicalkeyboard according to the first embodiment of the present invention. Thecontrol module 16 further includes a micro control unit 161, a keystrokedetermining unit 162, and an inductive control unit 163. The keystrokedetermining unit 162 is electrically connected to a contact (or referredto as key contact) corresponding to each key 12, and is configured to:detect whether the contact switches from a normally open state to aconductive state because of a keystroke, so as to generate a key scancode accordingly. The micro control unit 161 stores the key scan codeinto a second buffer 62 of keyboard buffer of the computing device 64 aby using a keyboard driver 56, and sends an interrupt request to aprocessor 66 of the computing device 64 a. Subsequently, the processor66 in the computing device 64 a reads the key scan code from the secondbuffer 62 in response to the interrupt request, to learn which key 12 isdepressed. The “keystroke” includes depressing (that is, Key Down,generating a “Make Code”) and releasing (that is, Key Up, generating a“Brake Code”) of a key. The forgoing is an acting procedure part of akeystroke on the physical keyboard in the present invention.

The inductive control unit 163 is electrically connected to the firstsensor 14, the second sensor 15, and the light source module 13. Theforegoing preset ambient luminance is stored in the inductive controlunit 163. When the ambient luminance detected by the first sensor 14 isless than the preset ambient luminance, the inductive control unit 163turns on the light source module 13 to provide a light source. When thedetected ambient luminance is equal to or greater than the presetambient luminance, the inductive control unit 163 turns on the secondsensor 15 to start to detect and obtain the operating position image.The micro control unit 161 stores the operating position image detectedand obtained by the second sensor 15 into a first buffer 60 of thekeyboard buffer of the computing device 64 a by using the keyboarddriver 56, and sends the interrupt request to the processor 66 in thecomputing device 64 a, to so that the processor 66 in the computingdevice 64 a can read the operating position image from the first buffer60 in response to the interrupt request. Subsequently, the imageanalyzing module 58 disposed on the computing device 64 a analyzes theimage data, to further obtain the operating position data, where theoperating position data includes a relative position data of a key 12which is referenced, that is, “the data of relative positions between afinger (or fingers) of a user and the keys 12”. When the operatingposition image cannot be identified and analyzed because the images areexcessively dark, the image analyzing module 58 notifies the microcontrol unit 161, so that the inductive control unit 163 can turn on thelight source module 13 to provide a light source or enhance the currentlight source. When the operating position data does not include “thedata of relative positions between the finger/fingers and the keys 12”,the image analyzing module 58 notifies the micro control unit 161, sothat the inductive control unit 163 can turn off the second sensor 15 tostop detecting and obtaining the operating position image.

The foregoing described buffers use the first-in first-out (FIFO) datastructure. In some embodiments, the operating position image is storedinto a dedicated first buffer 60, the processor 66 of the computingdevice 64 a is not notified by sending an interrupt request, andinstead, the processor 66 of the computing device 64 a automaticallyreads, in a polling manner and at intervals, the first buffer 60dedicated to the operating position image.

Similarly, after the key scan code is stored in a dedicated secondbuffer 62, the processor 66 of the computing device 64 a automaticallyreads, in a polling manner and at intervals, the second buffer 62dedicated to the key scan code.

Refer to FIG. 6. FIG. 6 is a system block diagram of a physical keyboard50 and a corresponding computing device 64 b according to a fifthembodiment of the present invention. A physical keyboard 50 of thisembodiment differs from the physical keyboard of a first embodiment inthat the physical keyboard 50 of this embodiment does not include anysensors, and functions and an operating flow of a keystroke determiningunit 162 thereof are as stated in the first embodiment, and thereforedetails are not described herein again. A micro control unit 161 hereinonly processes a key scan code generated by the keystroke determiningunit 162, and an operating flow thereof is also as stated in the firstembodiment. Moreover, the computing device 64 b mainly differs from thecomputing device 64 a of the first embodiment in that the computingdevice 64 b is provided with a fourth sensor 51 and a third buffer 63individual from a keyboard buffer of the keyboard driver 56. The fourthsensor 51 is configured to detect and obtain an operating positionimage. In this embodiment, the fourth sensor 51 is a camera (or a cameramodule). The operating position image detected and obtained by thefourth sensor 51 is a captured photo image of a finger (or fingers) andkeys 12. After the operating position image is stored into the thirdbuffer 63, the image is analyzed by an image analyzing module 58, so asto be converted into an operating position data, where the operatingposition data includes relative position data of these keys 12 which aretouched or approached (these two are referred to as “referenced keys”),i.e., relative position data of the finger (or fingers) of the user andone of these keys 12. The fourth sensor 51 may be, for example, abuilt-in camera above a screen of a notebook computer, and further forexample, a virtual reality device HTC Vive is provided a front-facingcamera, where both may be configured to capture the operating positionimage of the physical keyboard 50.

Refer to FIG. 7. FIG. 7 is a system block diagram of a physical keyboard50 and a corresponding computing device 64 c according to a sixthembodiment of the present invention. The physical keyboard 50 of thisembodiment is the same to that of the fifth embodiment, and thereforedetails are not described herein again. Moreover, the computing device64 c differs from the computing device 64 b of the fifth embodiment inthat a fifth sensor 52 and a third light source module 53 are added,where the fifth sensor 52 is configured to detect whether an ambientluminance of the plurality of keys 12 is sufficient, so that the fourthsensor 51 may detect and obtain required data. In this way, the fifthsensor 52 is a luminance sensor or called an ambient light sensor. Thethird light source module 53 may be an LED component, and may emitvisible lights. In other implementation aspects, the third light sourcemodule 53 may also be an LED component emitting infrared light. Thepresent invention is not limited thereto. When a processor 66 receivesand determines that the ambient luminance detected by the fifth sensor52 is equal to or greater than a preset ambient luminance, the fourthsensor 51 can be started and the operating position image of the fourthsensor 51 can be captured. When the processor 66 determines that theambient luminance detected by the fifth sensor 52 is smaller than thepreset ambient luminance, the processor 66 enables the third lightsource module 53 to provide a light source of the plurality of keys 12.

The following keyboard gesture instruction generating method and thecomputer program product and non-transitory computer readable storagemedium thereof according to the present invention are implemented byusing physical keyboards 10, 20, 30, 40, and 50 and computing devices 64a, 64 b, and 64 c of the foregoing embodiments. First, the followingkeyboard gesture instruction generating method and the computer programproduct thereof are described by using the physical keyboard 50 and thecomputing device 64 b of a fifth embodiment. The computer programproduct further includes a gesture instruction generating module 57 inaddition to the foregoing keyboard driver 56, and may be loaded by aprocessor 66 to be executed (as shown in FIG. 6). The gestureinstruction generating module 57 includes an instruction conditiondefinition table, so as to determine to which gesture instruction agesture event consists of multiple pieces of operating position data iscorresponding. In other words, the gesture instruction generating module57 is configured to read the multiple pieces of operating position data,so as to recognize the gesture event, and compare the gesture event withmultiple gesture instructions respectively including multipleinstruction conditions. If the gesture event matches with allinstruction conditions of one of the gesture instructions, the gestureinstruction is generated to be executed by the processor 66.

A keyboard gesture instruction is used to operate an applicationexecuted on the computing device 64 b, such as: closing the window,scrolling window content, spreading/pinching the window content,rotating the window content . . . .

The operating position data read by the gesture instruction generatingmodule 57 is generated by an image analyzing module 58 after recognizingand analyzing the operating position image. The operating position dataincludes relative position data of these keys 12 which are referenced,i.e., “relative position data of a finger (or fingers) of the user andthese keys 12”. The “relative position data of a finger (or fingers) andthese keys 12” is relative position data of keys 12 which are touched orapproached but not depressed by the finger (for example, the fingersuspends above the keys 12 at a very short distance). The “touched (orapproached but not depressed) keys 12” may be called “used keys 12” or“referenced keys 12”. Because a key gesture needs not to touch the keys12 of the physical keyboard 50 in an actual operation, an objective forthe finger to touch or approach the keys 12 is using a position of aphysical key 12 as a referential basis of a position when recognizing agesture, so as to reduce problems of erroneous determination whenrecognizing the gesture. The “referenced” keys 12 in the followingdescription refer to the keys 12 touched or approached by the finger ofthe user when operated by a gesture.

In some embodiments, the conditions of an instruction comprises aninitial quantity, a moving direction, an area, an initial position, amoving distance, a valid period, and a special condition. Each conditionis preset with a definition value (or referred to as a condition value)to be compared by the gesture instruction generating module 57 with acurrently obtained detection value. The initial quantity is the“quantity of referenced key 12 at an initial time”.

Because the gesture instruction may define a one-finger gestureoperation and a multiple-finger gesture operation, the quantity of thereferenced key 12 at the initial time needs to be distinguished. In someembodiments, the initial quantity of the instruction condition is onlydefined as two cases, “one-finger gesture” and “multiple-fingergesture”.

Therefore, the gesture instruction generating module 57 only needs todetermine whether a gesture event is performed by one finger or bymultiple fingers. For example, only key 12 of a “key J” is referenced, adetection value of the initial quantity is “1”, satisfying a one-fingercondition. If keys 12 of the “key J” and a “key K” are simultaneouslyreferenced, a detection value of the initial quantity is “2”, satisfyinga multiple-finger condition. If keys 12 of the “key J”, the “key K”, anda “key L” are simultaneously referenced, a detection value of theinitial quantity is “3”, also satisfying a multiple-finger condition. Insome embodiments, the initial quantity of the instruction condition maybe defined as multiple cases such as “one-finger gesture”, “two-fingergesture”, and “three-finger gesture”. For example, only key 12 of a “keyJ” is referenced, a detection value of the initial quantity is “1”,satisfying a one-finger condition. If keys 12 of the “key J” and a “keyK” are simultaneously referenced, a detection value of the initialquantity is “2”, satisfying a two-finger condition. If keys 12 of the“key J”, the “key K”, and a “key L” are simultaneously referenced, adetection value of the initial quantity is “3”, satisfying athree-finger condition. Other cases can be derived by analogy.

The moving direction is a “moving direction of movement of the gesture”.The moving direction is determined by using a first referenced key 12 asan origin (that is, an initial position), and by means of “a nextreferenced key 12” and “next two referenced keys 12”. In someembodiments, it is defined that of the moving direction of theinstruction condition is determined by means of two neighboring keys 12.A layout of a QWERTY-arranged keyboard is used as an example fordescription (examples below all use the same keyboard layout, but thepresent invention is not limited thereto). For example, key 12 of a “keyJ” is referenced, that is, the “key J” is considered as an origin, andthen key 12 of a “key K” neighboring to the “key J” is referenced, sothat the gesture instruction generating module 57 obtains a detectionvalue of “moving to the right”. In some embodiments, it is defined thatthe moving direction of the instruction condition is determined by meansof next two consecutive keys 12 which are neighboring to each other andare in the same direction. For example, the key 12 of the “key J” isreferenced, that is, the “key J” is considered as an origin, and then,the key 12 of the neighboring “key K” is referenced. In this case, thegesture instruction generating module 57 considers a detection value ofa first stage as “moving to the right”, and then the gesture instructiongenerating module 57 continue to perform tracking. Later, key 12 of aneighboring “key L” is referenced, that is, a detection value of asecond stage is also “moving to the right”.

The gesture instruction generating module 57 finally determines themoving direction as “moving to the right” until the third consecutivekey 12 is referenced, and detection values of moving directions of theintermediate two stages are the same. Tracing of each gesture eventtraced by the gesture instruction generating module 57 is suspended whenthe gesture event matches a comparison condition, or when a next key 12is not moved to when a preset valid period (for example, a preset validperiod is 500 milliseconds) is passed, or when a keystroke operationoccurs (which generating a key scan code), or when definition conditionsare violated (for example, a next referenced key 12 is not neighboringto a previously referenced key 12, or a moving direction of the secondstage is different from that of the first stage).

Determining of a one-finger moving direction is described above. Thereare two methods for determining a multiple-finger moving direction. Insome embodiments, if multiple keys 12 are simultaneously referenced,when fingers move, the gesture instruction generating module 57simultaneously monitors multiple gesture events. If a gesture event ofany finger matches a condition, a moving direction of the finger may bedetermined. For example, assuming it is defined that the movingdirection of the instruction condition is determined by next twoconsecutive keys 12 that are neighboring to each other and in the samedirection, when a user simultaneously refers keys 12 of the “key J”, the“key K”, and the “key L” respectively by using his first finger, middlefinger, and ring finger, the gesture instruction generating module 57considers the “key J”, the “key K”, and the “key L” as three origins andthree gesture events. Then, if keys 12 of the “key K”, the “key L”, anda “key;” are simultaneously referenced, in this case, a detection valueof a moving direction of the first stage is “rightward”. The threegesture events are continued to be traced. Later, if only keys 12 of the“key L” and the “key;” are simultaneously referenced, it may be that thefirst finger leaves halfway, or may be that the ring finger leaveshalfway. But whatever, a detection value of a moving direction of thesecond stage can still be considered as “rightward”, because for thegesture instruction generating module 57, a gesture event with key 12 ofthe “key J” as an origin matches “moving to next two consecutive keys 12that are neighboring to each other and in the same direction”, and isthereby determined as matching the condition. In some embodiments, whenmultiple keys 12 are simultaneously referenced, the gesture instructiongenerating module 57 considers moving in the same direction as the samegesture event having multiple origins. When the fingers move from theorigins to neighboring keys 12 that are referenced next, a detectionvalue of a moving direction of the first stage may be obtained, and thenthe fingers move to neighboring keys 12 that are referenced next, and adetection value of a moving direction of the second stage may beobtained. If detection values of the two stages are the same, thegesture instruction generating module 57 may determine the movingdirection.

The area is “a key section on the physical keyboard 50 where the gestureis operating”. Using a standard Windows keyboard as an example fordescription, definition values of the area includes alphanumeric keys,function keys, modifier keys, numeric keys, and a qwerty keyboard (thatincludes all the four parts). The alphanumeric keys include number keys,English letter keys, shift keys, symbolic keys, a space key, a backspacekey, a caps-lock key, a new line key (or referred to as an enter key),manipulation keys, and conversion keys. The function keys include twelvefunction keys located at the first row of the physical keyboard 50.

The numeric keys include a number-lock key, number keys, mathematicalsymbol keys, an enter key, and a decimal point key that are located atthe rightmost end of the physical keyboard 50. The modifier keys includearrow keys, an insert key, a delete key, a return key, an end key, askip key, a page-up key, and a page-down key. When an area condition ofan instruction is limited to a particular area, it represents that agesture event occurs in the particular area is determined to be valid.For example, if a “window operation closing instruction” is limited tobeing valid only in the numeric keys, and it would be invalid whenoccurs in the alphanumeric keys. Dividing areas to receive differentgesture instructions can effectively improve efficiency of monitoringgesture events by the gesture instruction generating module 57, reduceinstruction misjudgments, and lower hardware resource consumption of thecomputing device 64 b (such as computing resources of the processor 66).

The initial position refers to whether an origin (first referenced key12) needs to start from a designated particular key, for example, fromthe first three rows of the numeric keys, or from the middle two columnsof the alphanumeric keys, or from two outer sides of the alphanumerickeys, or from any key.

The moving distance is “the quantity of keys 12 that are in the samedirection and consecutively referenced” and is, for example, greaterthan or equal to (>=) three keys. If a condition of the moving distanceis defined excessively short (a definition value is excessively small),misjudgments by the gesture instruction generating module 57 may easilybe caused, but if the condition of the moving distance is definedexcessively long (the definition value is excessively large), a user mayprobably consider it is not easy for operation (for example, fingersslide to an edge of an area, but still a distance required in thecondition is not reach). The valid period refers to “a longest timeinterval for sliding two adjacent keys 12”, for example: when a presetdefined value of the valid period is 500 ms, and when a time forsuccessively sliding two adjacent referenced keys 12 exceeds a defaultdefined value, tracking of the gesture event is stopped. Herein, the“sliding” refers to that the finger “continuously touches” or“continuously approaches” (not necessarily touches the keys 12) adjacentkeys 12.

In some embodiments, the valid period may be determined by using timeintervals generated by two sets of operating position data havingdifferent relative position data, where the same set of operatingposition data comes from the same operating position image. Supposingthat in the operating position data analyzed by the image analyzingmodule 58, a set of operating position data having different relativeposition data is obtained every 300 ms, and if the defined value of thevalid period is 500 ms, two sets of operating position data successivein time are within the valid period.

The special condition is applicable to combinations of relativelycomplicated conditions, such as two groups (that is, two hands with eachone using at least one finger) moving in opposite directions, a distanceof two keys drawn near inwardly, a distance of two keys expandedoutwardly, etc.

Below is partial content of an instruction condition definition table ofan embodiment. Definition values in the instruction condition definitiontable are used for exemplary description but are not used to limit thepresent invention. Referring to Table 1, Table 1 is the instructioncondition definition table. A corresponding gesture instruction can begenerated only when all conditions in the field of “instructionconditions” are matched after comparison. If a definition value of acondition is “null” or “any key”, comparison may be neglected. If adefinition value of a condition is “horizontal”, it represents that anobtained detection value may be “leftward” or “rightward”. Content inthe field of “execution” is a job transferred to be executed by theprocessor 66 after a gesture instruction is generated. An “activewindow” is a window that is currently used, and “window content” is aview. For example, window content of a photo browsing program is a viewof a photo, and enlarging, contracting, or rotating window contentrepresents enlarging, contracting, or rotating the photo.

TABLE 1 Instruction conditions Execution Horizontal Initial quantity = 1To scroll a horizontal scrolling scrolling Moving direction: horizontalbar of an active window instruction Area: alphanumeric keys according toa moving direction Initial position: any key of a gesture instructionMoving distance: >=3 Valid period: <=500 ms Special condition: nullVertical Initial quantity = 1 To scroll a vertical scrolling barscrolling Moving direction: vertical of an active window accordinginstruction Area: alphanumeric keys to a moving direction of Initialposition: any key a gesture instruction Moving distance: >=3 Validperiod: <=500 ms Special condition: null Program Initial quantity: >1 Toswitch another program in switching Moving direction: horizontalexecution to be an active instruction Area: function keys window Initialposition: any key Moving distance: >=3 Valid period: <=500 ms Specialcondition: null Program Initial quantity >1 To close an active window(ends closing Moving direction: downward program) instruction Area:numeric keys Initial position: first three rows Moving distance: >=3Valid period: <=500 ms Special condition: null Window Initial quantity =1 To minimize an active window contraction Moving direction: downwardinstruction Area: numeric keys Initial position: first three rows Movingdistance: >=3 Valid period: <=500 ms Special condition: null WindowInitial quantity = 1 To maximize an active window enlarging Movingdirection: upward instruction Area: numeric keys Initial position: lastthree rows Moving distance: >=3 Valid period: <=500 ms Specialcondition: null Content Initial quantity: >1 To Enlarge (spread) contentof enlarging Moving direction: horizontal an active window instructionAreas: alphanumeric keys, not including keys on a row of a space keyInitial position: two columns of T and Y in the middle Movingdistance: >=3 Valid period: <=500 ms Special condition: a distance oftwo keys expanded outwardly Content Initial quantity: >1 To contract(pinch) content of contraction Moving direction: horizontal an activewindow instruction Areas: alphanumeric keys, not including a space keyInitial position: two outer sides Moving distance: >=3 Valid period:<=500 ms Special condition: a distance of two keys drawn near inwardlyRotate Initial quantity: >=4 To rotate content of an active instructionMoving direction: vertical window clockwise or Areas: alphanumeric keys,not counterclockwise according including a space key to a movingdirection of Initial position: any key a gesture instruction Movingdistance: >=3 Valid period: <=500 ms Special condition: two groupsmoving in vertically opposite directions

The “content enlarging instruction” in Table 1 is used as an example todescribe definition values of conditions of the content enlarging(spreading) instruction. The content enlarging instruction in thisembodiment needs to be operated by using at least one finger of eachhand, such as a first finger of a left hand and a first finger of aright hand, and therefore, an initial quantity is greater than 1. Amoving direction is horizontal, that is, the referenced keys 12 must bein the same row. A valid area is limited to alphanumeric keys but doesnot include a row of a space key. That is, only three rows of letterkeys and a row of number keys, altogether four rows may be operated. Aninitial position starts from columns T and Y in the middle. That is,initial keys need to be a “key 5” and a “key 6”, or a “key T” and a “keyY”, or a “key G” and a “key H”, or a “key B” and a “key N”, altogetherfour pairs may be used as initial positions. A moving distance isgreater than or equal to 3, that is, the content enlarging instructionis executed only when three neighboring keys are consecutivelyreferenced. A valid period is less than or equal to 500 ms, that is,time intervals of all referenced keys 12 need to be less than or equalto 500 ms. A special condition is a distance of two keys expandedoutwardly. Using the pair of the “key G” and the “key H” as an example,a first step is that the left first finger refers to “key G”, and theright first finger refers to the “key H”. A second step is that the leftfirst finger refers to a “key F”, and the right first finger refers to a“key J”. A third step is that the left first finger refers to a “key D”,and the right first finger refers to a “key K”. A fourth step is thatthe left first finger refers to a “key S”, and the right first fingerrefers to a “key L”. When content of an active window is a photo, in thefirst step, a size of the photo is unchanged. In the second step, thesize of the photo is still unchanged. In the third step, because themoving distance of the instruction condition is satisfied, the photo isenlarged by 25% of its original size, and in the fourth step, the photois continuously enlarged by 50% of its original size. The four steps inthe above-described example are consecutive actions, and time intervalsof touching keys do not exceed the definition value of the valid period.If a user intends to continuously enlarge the photo, after the fourthstep, the user may continuously expand the distance outwardly usingfingers, or start from the beginning and repeat operations from thefirst step to the fourth step.

Refer to FIG. 8. FIG. 8 is an operation flowchart of generating agesture instruction according to an embodiment of the present invention,and describes an operation flowchart of, after the gesture instructiongenerating module 57 loads the instruction conditions, reading theoperating position data and continuously comparing the default definedvalue, including the following.

Step S101: Load an instruction condition definition table. The gestureinstruction generating module 57 includes the instruction conditiondefinition table, and loads the instruction condition definition tableto memory of the computing device 64 b.

Step S102: Read operating position data. The operating position dataread by the gesture instruction generating module 57 is generated by theimage analyzing module 58 after recognizing and analyzing the operatingposition image. The operating position data includes relative positiondata of these keys 12 which are referenced, i.e., “the relative positiondata of the finger of the user and these keys 12”. See steps S202 toS204 of FIG. 21 for detailed description of the steps.

Step S103: Determine whether to move. The gesture instruction generatingmodule 57 determines whether the finger moves according to the readoperating position data, so as to decide whether to perform a continuouscomparison of a new gesture event. The operating position data convertedby the operating position image after being performed with an imagerecognition and analysis may include zero or multiple referenced keys12, so that whether the finger moves may be compared by using theoperating position data converted by “two operating position imagessuccessive in time”. The gesture instruction generating module 57determines, according to relative positions and a sequence in time ofthe referenced keys 12, whether the finger continuously slides twoadjacent keys within the valid period. If “two adjacent keys 12 arecontinuously slid within the valid period”, the data are considered tobe related, or may be referred as that “two sets of operating positiondata successive in time slide two adjacent keys 12”. In addition to thetwo sets of operating position data, subsequent several sets ofsuccessive operating position data included therein are temporarilyconsidered as “related”. The multiple related operating position dataare considered as the new gesture event. In other words, the gestureinstruction generating module 57 recognizes a gesture event according tomultiple successive operating position data. This gesture event isstarting tracked at this time, and multiple detected values of “theserelated operating position data” obtained by tracking are used tocontinuously compare defined values of various conditions. If the fingerdoes not move, then back to the former step S102 to read new operatingposition data again; and if the finger moves, step S104 is performed.

Step S104: Compare instruction conditions of a gesture event. When thegesture instruction generating module 57 determines that the fingermoves in the former step, a continuous comparison of the new gestureevent is turned on, so as to compare the multiple detected values of thepreviously obtained gesture event to be compared with the defined valuesof the instruction conditions. For example: when “multiple sets ofsuccessive operating position data” represents as a new gesture event,if an area to which referenced keys 12 in a first set of operatingposition data belong is the alphanumeric key, an area detected value ofthe gesture event is the “alphanumeric key”, and the referenced keys 12in the first set of operating position data are initial positions. Ifquantity of the referenced keys 12 in the first set of operatingposition data is one, a detected value of an initial quantity of thegesture event is “1”, etc.

Step S105: Determine whether an area is met. The gesture instructiongenerating module 57 narrows a comparison range by comparing thedetected values and the defined values within the area. For example:when the area detected value of the gesture event is the “alphanumerickey”, the following comparative job may ignore gesture instructionswhich are only allowed to occur at the numeric keys, the function keys,and the modifier keys, so as to only compare gesture instructions whichare only allowed to occur at the alphanumeric keys.

Perform step S115 if the area is not met. Perform step S106 if the areais met.

Step S106: Determine whether initial positions are met. The gestureinstruction generating module 57 compares the detected values and thedefined values using the initial positions of the gesture event. Performstep S115 if the initial positions are not met. Perform step S107 if theinitial positions are met.

Step S107: Determine whether a moving direction is met. The gestureinstruction generating module 57 compares the detected values and thedefined values using the moving direction of the gesture event. Performstep S115 if the moving direction is not met. Perform step S108 if themoving direction is met.

Step S108: Determine whether an initial quantity is met. The gestureinstruction generating module 57 compares the detected values and thedefined values using the initial quantity of the gesture event. Performstep S115 if the initial quantity is not met. Perform step S109 if theinitial quantity is met.

Step S109: Determine whether a moving distance is met. The gestureinstruction generating module 57 compares the detected values and thedefined values using the moving distance of the gesture event. Performstep S115 if the moving distance is not met. Perform step S110 if themoving distance is met.

Step S110: Determine whether a special condition is met. The gestureinstruction generating module 57 compares the detected values and thedefined values using the special condition of the gesture event. Performstep S115 if the special condition is not met. Perform step S111 if thespecial condition is met.

Step S111: Determine whether time intervals are within a valid period.In some embodiments, the time intervals generated by the operatingposition data are within the valid period. Therefore, the gestureinstruction generating module 57 only needs to verify whether theoperating position data included in the gesture event are datasuccessive in time, so as to determine whether the time intervals arewithin the valid period. Perform step S115 if the time intervals are notwithin the valid period. Perform step S112 if the time intervals arewithin the valid period.

Step S112: Determine whether there is a key scan code. The gestureinstruction generating module 57 checks the second buffer 62, andperforms step S115 if there is a key scan code. Step S113 is performedif there is no key scan code. While the gesture instruction generatingmodule 57 tracking the gesture event, if a keystroke occurs (generatingthe key scan code), it indicates that an objective of the fingertouching/approaching the keys 12 in the operating position data is notfor operating with a gesture but for typing data. Therefore, thetracking of the gesture event is stopped.

Step S113: Generate a gesture instruction. When the foregoingcomparisons of conditions from step S105 to step S112 are all met, thegesture instruction generating module 57 generates the gestureinstruction.

Step S114: Execute, by the processor 66, the gesture instruction. Afterthe gesture instruction is generated, the gesture instruction generatingmodule 57 delivers the gesture instruction to the processor 66 to beexecuted.

Step S115: Stop comparison of the gesture event, i.e., stop tracking ofthe gesture event.

The operational flowchart of generating a gesture instruction accordingto an embodiment is described above. The steps of condition comparisonmay be arranged in different sequences in different embodiments.Different instruction conditions used for comparison may be used inother different embodiments. The present invention is not limited to thesequence of the procedures and the instruction conditions describedabove. In some embodiments, a gesture instruction may split intomultiple phases (stages), for example, a starting phase, a moving phase,and an ending phase. The gesture event of such an instruction keepsbeing tracked until the ending phase, while the processor 66 isexecuting each phase of the instruction. So the processor 66 may beexecuting one of the phases of the instruction while the comparison jobsare still in-progress.

In some embodiments, the computer program product additionally includesan on-screen virtual keyboard which is alternatively referred to as anon-screen keyboard (OSK) in Microsoft Windows operating systems. Theon-screen keyboard in this embodiment is a computer program installed ona computing device 64 and executed by a processor 66, to display akeyboard layout, a first key mark, a second key mark, and a gestureoperation track. Referring to FIG. 9 to FIG. 13, FIG. 9 to FIG. 13 areschematic diagrams of an on-screen keyboard 68 a according to a seventhembodiment of the present invention. The on-screen keyboard 68 a of thisembodiment is utilizing the computing device 64 b and the physicalkeyboard 50 in the fifth embodiment for describing. FIG. 9 describesdisplaying of the on-screen keyboard 68 a of this embodiment on a screen76 a of a computing device 64 b. After execution, the on-screen keyboard68 a is located at a topmost layer (that is, a view attribute is set as“Topmost”) of an operating system desktop 70, that is, even if windowsof other applications are displayed at the same position of theon-screen keyboard 68 a, the windows still do not block displaying ofthe on-screen keyboard 68 a. A keyboard layout of the on-screen keyboard68 a is drawn corresponding to a layout of a physical keyboard 50 (shownin FIG. 11), to avoid that the on-screen keyboard 68 a is excessivelydifferent from real keys in positions resulting in that the user is notadapted to the on-screen keyboard 68 a. An active window 78 is athird-party application, and is a notepad herein. The on-screen keyboard68 a is a part of the computer program product.

FIG. 10 is an enlarged schematic diagram of the on-screen keyboard 68 aand describes that when fingers of a user are touching the keys 12 ofthe physical keyboard 50, or “approaching but not touching” the keys 12of the physical keyboard 50, the fourth sensor 51 is detecting andobtaining an operating position image which is a captured photo image ofa finger (or fingers) and keys 12. The operating position image is thenanalyzed by an image analyzing module 58, so as to be converted into anoperating position data, where the operating position data includesrelative position data of these keys 12 which are touched or approached(these two are referred to as “referenced keys 12”), i.e., relativeposition data of the finger (or fingers) of the user and one of thesekeys 12. The processor 66 further converts the operating position datainto a first key mark 72. The first key mark 72 is displayed at aposition on the on-screen keyboard 68 a which corresponds to thereferenced key 12. Herein, the marked positions of the first key marks72 include a “key A”, a “key S”, a “key D”, a “key F”, a “key J”, a “keyK”, a “key L”, a “key;” and a “space key”. FIG. 11 corresponds to FIG.10. FIG. 11 describes keys 12 of the “key A”, the “key S”, the “key D”,the “key F”, the “key J”, the “key K”, the “key L”, the “key;” and the“space key” on the physical keyboard 50 are being referenced by thefingers of the user. When the fingers of the user are referring to thekeys 12 but not depressing the keys, in this case, the positions of thefingers are referred to as the “fingers standby positions” or the“fingers ready positions”.

FIG. 12 describes that the “key J” on the on-screen keyboard 68 a ismarked with both the first key mark 72 and a second key mark 74, thatis, when the fingers of the user are touching or approaching the keys 12of the “key A”, the “key S”, the “key D”, the “key F”, the “key J”, the“key K”, the “key L”, the “key;” and the “space key” on the physicalkeyboard 50, and at the same time the finger of the user is depressingthe key 12 of the “key J”. In other words, not only the key 12 of the“key J” is touched or approached to generate an operating position data,but is also depressed to generate a key scan code (that is, a “MakeCode”). This case occurs when the key 12 of the “key J” is depresseddown (the “Make Code” is generated), but the key 12 of the “key J” isnot released (the “Break Code” is not yet generated). When typing isperformed at a normal speed, the second key mark 74 exists on theon-screen keyboard 68 a for a considerably short time (or referred to asa display time) which is usually less than one second or even less thanone tenth of a second, and is a flashing to human eyes.

Theoretically, it must be that the fingers touch (or approach) the keys12 first and then depress the keys 12. If typing is performed at arelatively fast speed, and the finger does not touch (or approach) thekey 12 until a keystroke, the operating position data and the key scancode (that is, a “Make Code”) of the same key are “almost” generatedsimultaneously (there is still a difference in time between theoperating position data and the key scan code, but the difference mightbe counted in milliseconds). That is, the first key mark 72 and thesecond key mark 74 of the same key are displayed “almost” simultaneously(human eyes are not easy to perceive a difference in time so as to feelthat the first key mark 72 and the second key mark 74 of the same keyare simultaneously displayed) on the on-screen keyboard 68 a.

However, when the majority persons (specifically those who type with twohands in correct fingering) type, before depressing keys (for example,when the persons are considering which word is to be used), the personsare accustomed to placing fingers at the “home keys” (or referred to asa Home Row, that is on the key 12 touched (or approached) by the fingersshown in FIG. 11), or placing fingers on a will-be-pressed key 12. Then,the key 12 is depressed. That is, sometimes, a first key mark 72 of akey 12 is displayed obviously earlier than a second key mark 74 of thesame key on the on-screen keyboard 68 a. In addition to that second keymarks 74 of a few locking keys (such as a “caps-lock key”, a“number-lock key”, and a “scroll-lock key” of a locking-key type or an“insert key” of a modifier key type) are separately displayed on theon-screen keyboard 68 a and do not coexist with the first key marks 72,as to keystrokes (which generate “Make Code”) on other keys, the firstkey marks 72 must be displayed earlier than the second key marks 74, orboth the first key marks 72 and the second key marks 74 are almostsimultaneously displayed on the on-screen keyboard 68 a, and it is notpossible that the second key marks 74 are separately displayed withoutthe first key marks 72. Visually, human eyes can obviously discriminatea difference between two overlapped marks and a single mark (using FIG.13 as an example). Although the second key mark 74 is only a flashing tothe human eyes, the second key mark 74 (shown in the top of FIG. 13)that exists separately is obvious different from two overlapped marks(shown in the bottom of FIG. 13), and specifically when the marks of twodifferent colors are displayed in an overlapped manner, the upper-layermark with incomplete transparency, will generate a visual effect of athird color. The human eyes can obviously distinguish a differencebetween the third color and colors of the two marks. Therefore, aprinciple that “the second key mark 74 cannot be separately displayed”can be used as an inspection mechanism visually. When a second key mark74 of a key is separately displayed on the on-screen keyboard 68 a, itrepresents “occurrence of an error” (for example, the operating positionimage taken by the camera is not clear, causing the image analyzingmodule 58 to misjudge), and a user may be reminded to take a notice. Insome embodiments, the on-screen keyboard 68 a has an automaticallyalerting mechanism. When a second key mark 74 of a key 12 is separatelydisplayed on the on-screen keyboard 68 a, the processor 66 calculates adifference between the misjudged position (where the first key mark 72incorrectly displayed) and the actual position (where the keystrokeoccurred). The calculated difference is then passed to the imageanalyzing module 58 and/or the processor 66 for correcting someparameters of generating the operating position data later on. This canreduce the probability of misjudgment occurs.

Herein the actual position is an individually displayed second key mark74, that is the second key mark 74 generated when a key 12 on thephysical keyboard 50 is depressed. The second key mark 74 is a correctmarker because a keystroke of the key 12 is really received. Theerroneously determined position refers to an adjacent first key mark 72which is not displayed, when depressed, in a manner of overlapping thesecond key mark 74. As stated above, when the obtained data is correct,the first key mark 72 (a position of a referenced key) needs to overlapthe second key mark 74 (the depressed key 12). Therefore, when the key12 is depressed, the first key mark 72 and the second key mark 74 of thekey 12 that are overlapped needs to be displayed. When only the secondkey mark 74 is displayed on the on-screen keyboard 68 a, the first keymark 72 may be marked at a wrong position because the operating positiondata obtained by analysis has an error. At this time, the wrongly markedposition of the first key mark 72 is defined as an erroneouslydetermined position.

Because the second key mark 74 is only a flashing, and in someembodiments, only the first key mark 72 is marked on the on-screenkeyboard 68 a to display a finger standby position without the secondkey mark 74, the on-screen keyboard 68 a directly omits processing ofthe key scan code, as shown in FIG. 10.

FIG. 13 is an enlarged schematic diagram of the “key J” on the on-screenkeyboard 68 a shown in FIG. 12 and describes two marks on the “key J”.The second key mark 74 in the top is marked in highlighted, and in anaspect, a square, a black background, and white words are used. Thefirst key mark 72 in the middle is marked in a color ofsemi-transparency and different shapes, and in an aspect, a circle andlight grey is used. A figure of overlapped marks in the bottom is anenlarged schematic diagram of the “key J” in FIG. 12. To obviouslydistinguish the first key mark 72 from the second key mark 74, in someembodiments, the first key mark 72 and the second key mark 74 are markedin different shapes, such as combinations of a circle and a square, asquare and a triangle, and a circle and a star. In some embodiments, thefirst key mark 72 and the second key mark 74 are marked in differentcolors, such as combinations of yellow and green, red and blue, blackand yellow, and light grey and red. In some embodiments, the first keymark 72 and the second key mark 74 are marked in different shapes anddifferent colors. In some embodiments, a mark marked at an upper layeris displayed in a manner of incomplete transparency, so that a mark at alower layer can be seen. For example, if the first key mark 72 islocated at the upper layer, and the second key mark 74 is located at thelower layer, the first key mark 72 is displayed in 50% of transparency.That is, the first key mark 72 or the second key mark 74 located at theupper layer has incomplete transparency. In some embodiments, if thesame key 12 is referenced (touched or approached) and depressed, the key12 may be marked with a third key mark of shapes and colors differentfrom those of the first key mark 72 and the second key mark 74.

FIG. 14 shows the marks displayed on the on-screen keyboard 68 a whenthe fingers of the user are touching or approaching the keys 12 of the“Shift key”, the “key S”, the “key D”, the “key F”, the “key J”, the“key K”, the “key L”, the “key;” and the “space key” on the physicalkeyboard 50 (generating the operating position image) and aresimultaneously depressing keys 12 of the “Shift key” and the “key J”(generating the key scan codes which are “Make Code”). This figuredescribes that after a key 12 of the “Shift key” is depressed and thekey scan code is generated, the key representation characters on theon-screen keyboard 68 a also change, that is, the key representationcharacters change from original lowercase to uppercase, and the keywhich has two key representation characters (such as numeric keys andsymbolic keys) change from the original lower part to the upper part.For example, the “key 1” of the numeric keys has two key representationcharacters which respectively are the upper part “!” and the lower part“1”, and the “key/” of the symbolic keys has two key representationcharacters which respectively are the upper part “?” and the lower part“/”. When the “Shift key” is not depressed, “1” and “/” at the lowerpart are displayed (shown in FIG. 10), but after the “Shift key” isdepressed, “!” and “?” at the upper part are displayed (shown in FIG.14).

FIG. 15 shows the marks displayed on the on-screen keyboard 68 a whenthe finger of the user has depressed the key 12 of the caps-lock key(generating a key scan code which includes a “Make Code” and a “BreakCode”, that is, the key 12 is released), and then are touching orapproaching the keys 12 of the “key A”, the “key S”, the “key D”, the“key F”, the “key J”, the “key K”, the “key L”, the “key;” and the“space key”, and is depressing the key 12 of the “key J” (generating akey scan code which is a “Make Code”). The figure describes that afterthe key 12 of the caps-lock key generates a key scan code, the keyrepresentation characters of the letter keys on the on-screen keyboard68 a change from original lowercase letters to uppercase letters.

The two figures above describe that key scan codes of the “Shift key”and the “caps-lock key” may enable the on-screen keyboard 68 a toreplace a part of key representation characters. Similarly, a“number-lock key” can also enable the on-screen keyboard 68 a to replacea part of key representation characters. That is, the on-screen keyboard68 a not only has a function of displaying the first key mark 72 and thesecond key mark 74, but also includes a function of interacting with thekey scan code.

Referring to FIG. 16 to FIG. 18, FIG. 16 to FIG. 18 are schematicdiagrams of an on-screen keyboard 68 b according to an eighth embodimentof the present invention. Display screens in this embodiment are screens76 b and 76 c of a virtual reality display device, and the screens 76 band 76 c are double-barrel-shaped (shown in FIG. 16) ordouble-circle-shaped (shown in FIG. 17). After a user puts the virtualreality display device on his head, his eyes see an image shown in FIG.18. In some embodiments, the virtual reality display device is acomputing device 64 b having the screen 76 b and 76 c on itself. Forexample, a Gear VR of Samsung uses a smartphone as a computing device 64b, and when the present invention is implemented, an on-screen keyboard68 b and a keyboard driver 56 are installed on the smartphone. In someembodiments, the virtual reality display device provides display imagesof the screens 76 b and 76 c and additionally needs to be incommunication connection with a computing device 64 b for installingsoftware thereon and providing hardware computing resources. Forexample, HTC Vive uses a personal computer as the computing device 64 b,and when the present invention is implemented, an on-screen keyboard 68b and a keyboard driver 56 are installed on the personal computer. Insome embodiments, a keyboard layout is displayed in an effect ofnon-transparency on screens 76 a, 76 b, and 76 c. In some embodiments, akeyboard layout is displayed in an effect of incomplete transparency(translucent) on screens 76 a, 76 b, and 76 c. In some embodiments, auser may set transparency of a keyboard layout by himself/herself.Herein, the on-screen keyboard 68 b displays a keyboard layout in amanner of incomplete transparency, so that when the user uses theon-screen keyboard 68 b, the user still sees a background image (avirtual reality image). If the user is not adept at touch-typing, whenthe user cannot stare at the physical keyboard 50 (for example, the useris mounted with a virtual reality display device on his head), the usercan clearly know a current finger standby position (by means of thefirst key mark 72), and which key is depressed (by means of the secondkey mark 74) by using the on-screen keyboard 68 b, facilitating typingof data for the user.

Referring to FIG. 19, FIG. 19 is a schematic diagram of an on-screenkeyboard 68 c according to a ninth embodiment of the present invention.Differences between the figure and the seventh embodiment and the eighthembodiment lie in that the figure is a keyboard layout corresponding toa physical keyboard of a Mac (not shown). Methods for displaying thefirst key mark 72 and the second key mark 74 and interacting with thekey scan code by the on-screen keyboard 68 c in this embodiment toreplace a part of key representation characters are the same as those inthe above-described embodiments, and details are not repeatedlydescribed.

Referring to FIG. 20, FIG. 20 is a schematic diagram of an on-screenkeyboard 68 d according to a tenth embodiment of the present invention.A difference between this embodiment and the seventh embodiment lies inthat a keyboard layout of the on-screen keyboard 68 d is divided intotwo parts, a left part and a right part. That is, this embodimentcorresponds to a physical keyboard (not shown) separated in a left and aright part. Marks of a first key mark 72 and a second key mark 74 ofthis embodiment are the same as those of the above-describedembodiments, and details are not repeatedly described. Herein, acorresponding physical keyboard has 105 keys (that is, 104 standard keysof Windows and a duplicate space key). In some embodiments, a keyboardlayout of the physical keyboard is “80% scheme” (that is, rightmostnumeric keys are omitted) and has 88 keys, and a keyboard layout of theon-screen keyboard separated with the left and right part (not shown)corresponding to the physical keyboard is also “80% scheme”. In someother embodiments, a keyboard layout of the physical keyboard may be akeyboard separated by a left and a right part and is “75% scheme” or“80% scheme”, or may be a keyboard separated by a left and a right partof a Mac. A corresponding on-screen keyboard uses the same keyboardlayout as that of the physical keyboard.

Refer to FIG. 21. FIG. 21 is a flowchart of marking, during a useroperation process, the first key mark 72 and the second key mark 74 inthe on-screen keyboard 68 a according to an embodiment of the presentinvention; and describes a operating procedure of generating theoperating position image and key scan code by using the computing device64 b and the physical keyboard 50 of the fifth embodiment, andgenerating the corresponding first key mark 72 and second key mark 74 onthe on-screen keyboard 68 a, including:

Step S201: Touch or approach, by the finger, a key. The finger of theuser touches or approaches the key 12 of the physical keyboard 50, butdoes not depress the key, as shown in FIG. 11. The referenced keys 12herein are “key A”, a “key S”, a “key D”, a “key F”, a “key J”, a “keyK”, a “key L”, a “key”, and a “space key”.

Step S202: Generate the operating position image. When the fingertouches or approaches the key 12 of the physical keyboard 50 in theformer step, a fourth sensor 51 detects and obtains the operatingposition image, where the operating position image is an photo image ofthe finger (or fingers) and the key 12 on the physical keyboard 50.

Step S203: Store the operating position image to a third buffer 63. Whenthe fourth sensor 51 generates the operating position image in theformer step, the processor 66 stores the operating position image to thethird buffer 63.

Step S204: Analyze, by an image analyzing module 58, the operatingposition image. After the operating position image is stored to thethird buffer 63 in the former step, there are two embodiments for theprocessor 66 to read the operating position image from the third buffer63. One embodiment is that the fourth sensor 51 sends an interruptrequest to the processor 66 of the computing device 64 b, and then theprocessor 66 of the computing device 64 b responds to the interruptrequest to read the operating position image from the third buffer 63;the other embodiment is that the processor 66 of the computing device 64b automatically reads the operating position image from the third buffer63 at intervals in a polling manner. That is, an action of reading theoperating position image may be triggered in an interruption or apolling manner. After the processor 66 reads the operating positionimage, an image analyzing module 58 analyzes the operating positionimage, so as to convert the operating position image into an operatingposition data, where the operating position data includes the relativeposition data of these keys 12 which are referenced, i.e., relativeposition data of the finger (or fingers) of the user and these keys 12.

Step S205: Display the first key mark 72 at a position of a key on theon-screen keyboard 68 a which corresponds to the operating positiondata. After the image analyzing module 58 converts the operatingposition image into the operating position data in the former step, theprocessor 66 further converts the operating position data into the firstkey mark 72. That is, the first key mark 72 is displayed at a positionon the on-screen keyboard 68 a which corresponds to the referenced key12. Herein, positions of the corresponding operating position image are:a “key a”, a “key s”, a “key d”, a “key f”, a “key j”, a “key k”, a keyl”, a “key;”, and a “space key”, as shown in FIG. 10. The on-screenkeyboard 68 a is displayed at a topmost layer of the displayed image(i.e., a view property is set as “Topmost”). Generally, the on-screenkeyboard 68 a is displayed after the operating position data or the keyscan code is read, but may disappear after an idle timeout after beingdisplayed. Therefore, in some cases, although the operating positiondata or the key scan code is not read, the on-screen keyboard 68 a isstill in a display state and disappears after the idle timeout (if theoperating position data or the key scan code is not read during theperiod). In some embodiments, the on-screen keyboard 68 a may also bedisplayed when an application program is executed, keeps in the displaystate, and disappears until the user ends the application program. Insome embodiments, the on-screen keyboard 68 a is not immediatelydisplayed when the application program is executed, and displaying anddisappearance thereof are controlled by the user.

Step S206: Determine whether the finger leaves the key. The imageanalyzing module 58 may determine whether the finger leaves thereferenced key 12 according to the operating position data. That is, inthe analyzed operating position data, if only the key rather than therelative position data of the finger and the key is included, itindicates that the finger leaves the original referenced key 12. If afinger leaves the referenced key 12 (i.e., the corresponding key 12 ofthe generated first key mark 72 in the former step), step S207 isperformed; and if the finger still does not leave the referenced key 12,step S208 is performed.

Step S207: Cancel displaying the first key mark 72 on the on-screenkeyboard 68 a. When the image analyzing module 58 determines that afinger leaves “the corresponding key 12 of the generated first key mark72”, the on-screen keyboard 68 a cancels displaying the correspondingfirst key mark 72.

Step S208: Depress, by the finger, the key. When the finger depressesthe key 12, a key contact of the depressed key 12 changes from anormally open state into a conductive state because of the keystroke.

Step S209: Generate the key scan code. When the keystroke determiningunit 162 detects that a key contact of a key 12 changes from a normallyopen state into a conductive state, the key scan code (a key scan codeof the depressed key 12) is generated. Herein, using the “key J” beingdepressed as an example, the keystroke determining unit 162 generates akey scan code (being a Make Code) of the “key J”.

Step S210: Store the key scan code to the second buffer 62. After thekeystroke determining unit 162 generates the key scan code (Make Code)in the former step, the micro control unit 161 sends out the key scancode to the computing device 64 b, and the computing device 64 b storesthe key scan code (Make Code) of the “key J” to the second buffer 62 ofthe computing device 64 b by using a keyboard driver 56.

Step S211: Read, by the processor 66, the key scan code. After the microcontrol unit 161 stores the key scan code (Make Code) to the secondbuffer 62 in the former step, there are two embodiments for theprocessor 66 to read the key scan code from the second buffer 62. Oneembodiment is that the micro control unit 161 sends an interrupt requestto the processor 66 of the computing device 64 b, and then the processor66 of the computing device 64 b responds to the interrupt request toread the key scan code from the second buffer 62; the other embodimentis that the processor 66 of the computing device 64 b automaticallyreads the key scan code from the second buffer 62 at intervals in apolling manner. That is, an action of reading the key scan code may betriggered in an interruption or a polling manner.

Step S212: Display the second key mark 74 at a position of a key on theon-screen keyboard 68 a which corresponds to the key scan code. Afterthe processor 66 reads the key scan code (Make Code) in the former step,the processor 66 displays, according to the key scan code, the secondkey mark 74 on the on-screen keyboard 68 a at a position of a keycorresponding to the depressed key 12. Herein, the positioncorresponding to the key scan code is the “key j”. The displayed secondkey mark 74 and the previously displayed first key mark 72 are shown inFIG. 12.

Step S213: Cancel displaying the corresponding second key mark 74 on theon-screen keyboard 68 a. When the keystroke determining unit 162 detectsthat the key contact of the pressed key 12 changes from a conductivestate into a normally open state, the key scan code (a key scan code ofa released key 12) is generated. Herein, using the “key J” beingreleased as an example, the keystroke determining unit 162 generates akey scan code (being a Break Code) of the “key J”. After the key 12 isreleased, the computing device 64 b stores the key scan code (BreakCode) of the “key J” to the second buffer 62 by using the keyboarddriver 56. Subsequently, the processor 66 reads the key scan code (BreakCode) of the “key J”. Finally, the according to the key scan code (BreakCode) of the “key J”, displaying the corresponding second key mark 74 onthe on-screen keyboard 68 a is cancelled, as shown in FIG. 10 (supposingthat the finger of the user still touches or approaches the key 12 ofthe “key J” after the “key J” is released). Particularly, the processor66 may additional determine that the depressed key is a general key suchas a letter or a digit, or is a locking-key type such as the “Shift key”stated above. If the key scan code of the corresponding released key 12is received and the key 12 is not a locking-key type, displaying thecorresponding second key mark 74 is cancelled. On the contrary, if thekey scan code of the corresponding released key 12 is received and thekey 12 is a locking-key type, displaying the corresponding second keymark 74 is maintained. Displaying the corresponding second key mark 74is cancelled until the same second key mark 74 of the same key 12 whichcorresponds to the keystroke is received at a next time.

The foregoing is a method for marking the first key mark 72 and thesecond key mark 74 on the on-screen keyboard 68 a according to anembodiment of the present invention. The present invention further, insome embodiments, after step S207, supposes that the finger of the usercompletely leaves all keys 12 for an idle timeout, i.e., the processor66 of the computing device 64 b does not detect (i.e., read), before theidle timeout, the operating position data including relative data of thefinger of the user and these keys 12 and the key scan code, theon-screen keyboard 68 a is automatically hidden (not displayed on ascreen 76 a). On the contrary, if the original on-screen keyboard 68 ais not displayed on the screen 76 a, the on-screen keyboard 68 a isautomatically displayed on a screen 76 a after the finger of the usertouches any key 12. This embodiment is intended to hide the on-screenkeyboard 68 a when the user needs not to operate/type data using akeyboard, so as to avoid taking up an operating system desktop 70,thereby enabling the user to operate other application programs by usingcontrol operation devices, for example, operating a web page browserwith a mouse or a trackball to view web page, or playing a computer gamewith a joystick.

The first key mark 72 on the foregoing on-screen keyboards 68 a, 68 b,68 c, and 68 d are used to represent a static finger standby position,and in some embodiments, are further used to represent a dynamic gestureoperation track, where the “gesture operation track” refers to anoperation track of the finger at a gesture event on the physicalkeyboard 50.

Refer to FIG. 22. FIG. 22 is a schematic diagram of the on-screenkeyboard 68 a and the gesture operation track according to an embodimentof the present invention, where the upper on-screen keyboard 68 a andthe lower physical keyboard 50 in the figure are not drawn according toan actual proportion. This indicates that when the user issues a gestureinstruction at the physical keyboard 50, the gesture instruction hereinis the “window zooming-out instruction” (please refer to the instructioncondition definition table in the above embodiments), the user slidesdown with a single finger (only touching or approaching but notdepressing the keys) at three front lines of the numeric keys, and thegesture instruction generating module finally generates the windowcontraction instruction and delivers the instruction to the processor 66to be executed (detailed flow is stated above, and therefore details arenot described herein again), so as to minimize the active window 78. Atthis time, the corresponding first key mark 72 presented on theon-screen keyboard 68 a represents a gesture operation track of thegesture event of “window contraction instruction”.

The foregoing method for generating a gesture instruction, the methodfor displaying the on-screen keyboard 68 a, and the computer programproduct are described by using the physical keyboard 50 and thecomputing device 64 b of the fifth embodiment as examples, where thefourth sensor 51 detecting and obtaining the operating position image isa camera. When the foregoing methods and the computer program productare implemented by using the physical keyboard 10 of a first embodimentor the physical keyboard 20 and the computing device 64 a of a secondembodiment, the operating position image is detected and obtained by thesecond sensor 15. When the foregoing methods and the computer programproduct are implemented by using the physical keyboard 30 of the thirdembodiment or the physical keyboard 40 and the computing device 64 a ofa fourth embodiment, the operating position image is detected andobtained by the second sensor 36 and the third sensor 37.

When the foregoing methods and the computer program product areimplemented by using the physical keyboard 50 and the computing device64 c of a sixth embodiment, the operating position image is detected andobtained by the fourth sensor 51.

The computer program product of the present invention may include anon-screen keyboard 68 a, a keyboard driver 56, a gesture instructiongenerating module 57, a library (not shown), and/or an applicationprogramming interface (APIs, not shown). A part of or all of theoperation procedures described above, may be encapsulated as thekeyboard driver, the library, the application programming interface or adevice driver (e.g. a driver of a virtual reality display device), so asto be further used by other applications. The computer program productmay be stored in a computer readable recording medium, such as a HardDisk Drive, a Solid State Drive, a CD, a DVD, a USB flash disk, a memorycard, or recording media of other specifications, and can be loaded andexecuted by a computer (that is, the computing device 64 b describedabove).

In various embodiments of the present invention, program instructionsexecuted by a computing device can implement every block in theflowchart, a combination of the blocks in the flowchart, or the steps inevery embodiment. The program instructions are provided to a processorso as to be executed by the processor to thereby produce resourcesrequired for a machine or coordinated operation of hardware andsoftware; hence, the instructions are executed on the processor tothereby produce elements required for carrying out actions specified bythe blocks in the flowchart or technical effects. Different combinationsof program instructions also allow simultaneous execution of at leastsome operating steps specified in the blocks in the flowchart, whereasthe technical solutions indicated by the first through fourth programinstructions of an application may vary from embodiment to embodiment.Furthermore, it is also practicable that some of the steps are executedon one or more processors, for example, in the case of a computingdevice of multiple processors or in the case of coordinated operation ofa microprocessor in a mobile communication device and a peripheralinterface processor. Moreover, in the flowchart, at least one block or acombination of blocks may be performed along with the other block or theother combination of blocks simultaneously or performed in a sequencedifferent from its specified counterpart, without departing from thespirit and scope of the present invention.

Therefore, the blocks in the flowchart of the present invention supporta combination of elements required for performing specified actions ortechnical solutions, a combination of steps required for performingspecified actions or technical solutions, and program instructionelements required for performing specified actions or technicalsolutions. What is also understandable is that the specified actions ortechnical solutions are effectuated by a combination of blocks in theflowchart as well as each block in the flowchart of the presentinvention through the coordinated operation of a special-purposehardware-style system or special-purpose hardware, and programinstructions.

In view of the above, a keyboard gesture instruction of the presentinvention may reduce problems of erroneous determination whenrecognizing a gesture because a key position on the physical keyboard istaken as a referential basis of gesture comparison. In this way, theuser experience is improved, so as to satisfy requirements of the userfor operating a system by using a gesture instruction of the physicalkeyboard.

Although the present invention has been described in considerable detailwith reference to certain preferred embodiments thereof, the disclosureis not for limiting the scope of the invention. Persons having ordinaryskill in the art may make various modifications and changes withoutdeparting from the scope and spirit of the invention. Therefore, thescope of the appended claims should not be limited to the description ofthe preferred embodiments described above.

What is claimed is:
 1. A keyboard gesture instruction generating method,applied to a computing device connected to a physical keyboard, a screenand a sensor, the physical keyboard comprising a plurality of keys, andthe sensor being configured to detect and obtain an operating positionimage, wherein the keyboard gesture instruction generating methodcomprises: loading an instruction condition definition table, whereinthe instruction condition definition table comprises a plurality ofgesture instructions, and each gesture instruction comprises a pluralityof instruction conditions, and each instruction condition has acorresponding defined value; reading the operating position imageobtained by the sensor; converting the operating position image into anoperating position data, wherein the operating position data comprisesrelative position data of the keys which are referenced; recognizing agesture event according to multiple sets of the successive operatingposition data; obtaining a plurality of detected values of the gestureevent and comparing the detected values with these defined values; ifthese detected values match with the defined values of all theinstruction conditions of one of the gesture instructions, stoppingtracking of the gesture event; and executing the matching gestureinstruction.
 2. The keyboard gesture instruction generating methodaccording to claim 1, wherein the sensor is a camera.
 3. The keyboardgesture instruction generating method according to claim 1, wherein thephysical keyboard further responds to a keystroke on the key, so as tooutput a key scan code to the computing device, and the keyboard gestureinstruction generating method further comprises: reading the key scancode outputted by the physical keyboard and corresponding to thekeystroke on the key; and stopping comparing the gesture event if thekey scan code is read.
 4. The keyboard gesture instruction generatingmethod according to claim 1, wherein one of these instruction conditionsis an area of the physical keyboard, and the defined value of this areais alphanumeric keys, function keys, numeric keys, modifier keys, or aqwerty keyboard.
 5. The keyboard gesture instruction generating methodaccording to claim 1, wherein the screen displays an on-screen keyboard,and the keyboard gesture instruction generating method furthercomprises: displaying a first key mark at a position on the on-screenkeyboard of the screen which corresponds to the referenced keys.
 6. Thekeyboard gesture instruction generating method according to claim 1,wherein these gesture instructions at least comprise a horizontalscrolling instruction, a vertical scrolling instruction, a programswitching instruction, a program closing instruction, a windowcontraction instruction, a window enlarging instruction, a contentenlarging instruction, a content contraction instruction, or a rotationinstruction, wherein the horizontal scrolling instruction enables ahorizontal scrollbar of an active window displayed on the screen toscroll towards a moving direction of the gesture instruction; thevertical scrolling instruction enables a vertical scrollbar of theactive window to scroll towards the moving direction of the gestureinstruction; the program switching instruction switches a program inexecution to be the active window; the program closing instructioncloses the active window; the window contraction instruction minimizesthe active window; the window enlarging instruction maximizes the activewindow; the content enlarging instruction enlarges a content in theactive window; the content contraction instruction contract the contentin the active window; and the rotation instruction enables the contentin the active window to rotate clockwise or anticlockwise according tothe moving direction of the gesture instruction.
 7. A computer programproduct, comprising a plurality of computer executable instructionsstored in a non-transitory computer readable storage medium, wherein thecomputer executable instructions are loaded and executed by a computingdevice connected to a physical keyboard, a screen and a sensor to causethe computing device to implement a method for generating a keyboardgesture instruction, the physical keyboard comprising a plurality ofkeys, and the sensor being configured to detect and obtain an operatingposition image, wherein the method for generating the keyboard gestureinstruction comprises: loading an instruction condition definitiontable, wherein the instruction condition definition table comprises aplurality of gesture instructions, and each gesture instructioncomprises a plurality of instruction conditions, and each instructioncondition has a corresponding defined value; reading the operatingposition image obtained by the sensor; converting the operating positionimage into an operating position data, wherein the operating positiondata comprises relative position data of the keys which are referenced;recognizing a gesture event according to multiple sets of the successiveoperating position data; obtaining a plurality of detected values of thegesture event and comparing the detected values with these definedvalues; if these detected values match with the defined values of allthe instruction conditions of one of the gesture instructions, stoppingtracking of the gesture event; and executing the matching gestureinstruction.
 8. The computer program product for generating the keyboardgesture instruction according to claim 7, wherein the sensor is acamera.
 9. The computer program product for generating the keyboardgesture instruction according to claim 7, wherein the physical keyboardfurther responds to a keystroke on the key, so as to output a key scancode to the computing device, and the keyboard gesture instructiongenerating method further comprises: reading the key scan code outputtedby the physical keyboard and corresponding to the keystroke on the key;and stopping comparing the gesture event if the key scan code is read.10. The computer program product for generating the keyboard gestureinstruction according to claim 7, wherein one of these instructionconditions is an area of the physical keyboard, and the defined value ofthis area is alphanumeric keys, function keys, numeric keys, modifierkeys, or a qwerty keyboard.
 11. The computer program product forgenerating the keyboard gesture instruction according to claim 7,wherein the screen displays an on-screen keyboard, and the keyboardgesture instruction generating method further comprises: displaying afirst key mark at a position on the on-screen keyboard of the screenwhich corresponds to the referenced keys.
 12. The computer programproduct for generating the keyboard gesture instruction according toclaim 7, wherein these gesture instructions at least comprise ahorizontal scrolling instruction, a vertical scrolling instruction, aprogram switching instruction, a program closing instruction, a windowcontraction instruction, a window enlarging instruction, a contentenlarging instruction, a content contraction instruction, or a rotationinstruction, wherein the horizontal scrolling instruction enables ahorizontal scrollbar of an active window displayed on the screen toscroll towards a moving direction of the gesture instruction; thevertical scrolling instruction enables a vertical scrollbar of theactive window to scroll towards the moving direction of the gestureinstruction; the program switching instruction switches a program inexecution to be the active window; the program closing instructioncloses the active window; the window contraction instruction minimizesthe active window; the window enlarging instruction maximizes the activewindow; the content enlarging instruction enlarges a content in theactive window; the content contraction instruction contract the contentin the active window; and the rotation instruction enables the contentin the active window to rotate clockwise or anticlockwise according tothe moving direction of the gesture instruction.
 13. The computerprogram product for generating the keyboard gesture instructionaccording to claim 7, wherein the computer executable instructions areencapsulated as a library, an application programming interface, akeyboard driver or a driver of a virtual reality display device.
 14. Anon-transitory computer readable storage medium, storing a computerprogram, wherein the computer program comprises a plurality of computerexecutable instructions that, when executed by a computing deviceconnected to a physical keyboard, a screen and a sensor to cause thecomputing device to implement a keyboard gesture instruction generatingmethod, the physical keyboard comprising a plurality of keys, and thesensor being configured to detect and obtain an operating positionimage, wherein the keyboard gesture instruction generating methodcomprises: loading an instruction condition definition table, whereinthe instruction condition definition table comprises a plurality ofgesture instructions, and each gesture instruction comprises a pluralityof instruction conditions, and each instruction condition has acorresponding defined value; reading the operating position imageobtained by the sensor; converting the operating position image into anoperating position data, wherein the operating position data comprisesrelative position data of the keys which are referenced; recognizing agesture event according to multiple sets of the successive operatingposition data; obtaining a plurality of detected values of the gestureevent and comparing the detected values with these defined values; ifthese detected values match with the defined values of all theinstruction conditions of one of the gesture instructions, stoppingtracking of the gesture event; and executing the matching gestureinstruction.
 15. The non-transitory computer readable storage mediumaccording to claim 14, wherein the sensor is a camera.
 16. Thenon-transitory computer readable storage medium according to claim 14,wherein the physical keyboard further responds to a keystroke on thekey, so as to output a key scan code to the computing device, and thekeyboard gesture instruction generating method further comprises:reading the key scan code outputted by the physical keyboard andcorresponding to the keystroke on the key; and stopping comparing thegesture event if the key scan code is read.
 17. The non-transitorycomputer readable storage medium according to claim 14, wherein one ofthese instruction conditions is an area of the physical keyboard, andthe defined value of this area is alphanumeric keys, function keys,numeric keys, modifier keys, or a qwerty keyboard.
 18. Thenon-transitory computer readable storage medium according to claim 14,wherein the screen displays an on-screen keyboard, and the keyboardgesture instruction generating method further comprises: displaying afirst key mark at a position on the on-screen keyboard of the screenwhich corresponds to the referenced keys.
 19. The non-transitorycomputer readable storage medium according to claim 14, wherein thesegesture instructions at least comprise a horizontal scrollinginstruction, a vertical scrolling instruction, a program switchinginstruction, a program closing instruction, a window contractioninstruction, a window enlarging instruction, a content enlarginginstruction, a content contraction instruction, or a rotationinstruction, wherein the horizontal scrolling instruction enables ahorizontal scrollbar of an active window displayed on the screen toscroll towards a moving direction of the gesture instruction; thevertical scrolling instruction enables a vertical scrollbar of theactive window to scroll towards the moving direction of the gestureinstruction; the program switching instruction switches a program inexecution to be the active window; the program closing instructioncloses the active window; the window contraction instruction minimizesthe active window; the window enlarging instruction maximizes the activewindow; the content enlarging instruction enlarges a content in theactive window; the content contraction instruction contract the contentin the active window; and the rotation instruction enables the contentin the active window to rotate clockwise or anticlockwise according tothe moving direction of the gesture instruction.
 20. The non-transitorycomputer readable storage medium according to claim 14, wherein thecomputer executable instructions are encapsulated as a library, anapplication programming interface, a keyboard driver or a driver of avirtual reality display device.